3D Reconstruction of Non-Rigid Surfaces in Real-Time Using Wedge Elements

We present a new FEM (Finite Element Method) model for the 3D reconstruction of a deforming scene using as sole input a calibrated video sequence. Our approach extends the recently proposed 2D thin-plate FEM+EKF (Extended Kalman Filter) combination. Thinplate FEM is an approximation that models a deforming 3D thin solid as a surface, and then discretizes the surface as a mesh of planar triangles. In contrast, we propose a full-fledged 3D FEM formulation where the deforming 3D solid is discretized as a mesh of 3D wedge elements. The new 3D FEM formulation provides better conditioning for the rank analysis stage necessary to remove the rigid boundary points from the formulation. We show how the proposed formulation accurately estimates deformable scenes from real imagery even for strong deformations. Crucially we also show, for the first time to the best of our knowledge, NRSfM (Non-Rigid Structure from Motion) at 30Hz real-time over real imagery. Real-time can be achieved for our 3D FEM formulation combined with an EKF resulting in accurate estimates even for small size maps.